Temporal Events in All Dimensions and Scales

This paper describes a new representation for the audio and visual information in a video signal. We reduce the dimensionality of the signals with singular-value decompo-sitions (SVD) or mel-frequency cepstral coefficients (MFCC). We apply these transforms to word, (word transcript , semantic space or latent semantic indexing), image (color histogram data) and audio (timbre) data. Using scale-space techniques we find large jumps in a video's path, which are evidence for events. We use these techniques to analyze the temporal properties of the audio and image data in a video. This analysis creates a hierarchical segmentation of the video, or a table-of-contents, from both audio and the image data. 1 Browsing videotapes of image and sound (hereafter referred to as " videos ") is difficult. Often, there is an hour or more of material, and there is no roadmap to help viewers find their way through the medium. It would be tremendously helpful to have an automated way to create a hierarchical table of contents that listed major topic changes at the highest level, with subsegments down to individual shots. Realization of such an automated analysis requires the development of algorithms which can detect changes in the video and/or semantic content of a video as a function of time. We propose a technology that performs this indexing task by representing the all the major sources of data — images, non-speech audio and words — from the video into a common representations. This paper describes an approach to event detection based on dimensionality reduction and scale-space seg-mentation. We are looking for events marked by large changes in the audio or image data at many different time scales. We demonstrate this idea based on the semantic information in the video's transcript, color information in the sequence of images, and acoustic information in a musical accompaniment. Our technique is analogous to one that detects edges in an image. Instead of trying to find similar regions of the multimedia signal, called segments, we think of the audio– visual content as a signal and look for " large " changes in this signal or peaks in its derivative. The location of these changes are events; they represent the nodes in a table of contents. The techniques we describe in this paper allow us to characterize the temporal properties of both the audio and image data in the video. The color information in the image signal and …